To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation
summary
The gist
Multimodal large language models increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding <ref:2610.21613#pg6> The results recast adaptive Ref-AVS reasoning as a…
In short
The paper addresses how large multimodal models use reasoning during audio-visual tasks. It proposes a method to dynamically decide whether a model needs more reasoning by examining an internal state called 'honset' before any reasoning is generated. This allows the system to prune unnecessary thinking, reducing token usage by 60% while maintaining high accuracy.
Key concepts
- Honset
- This is a specific internal hidden state of the multimodal model captured right at the beginning of generation. It has already processed video, audio, and text evidence but contains no actual reasoning steps yet. This state encodes crucial information about whether the model needs further reasoning to complete its task.
- Pre-decisional Representation
- This concept suggests that the model's internal state before it starts generating a long chain of thought is already a 'pre-decisional representation.' It means the necessary decision about how much reasoning is required has already been made by integrating all available input evidence, rather than waiting to generate tokens to make that decision.
- Reasoning Budget Utility
- This refers to the value or utility of using reasoning steps for a specific query. The paper aims to predict this utility before the first token is generated using a single linear classifier trained on the honset state, enabling cost-aware routing of reasoning.
- Overthinking Trap
- This occurs when forcing a model through an overly long chain-of-thought for simple inputs leads to errors. The system identifies this trap by observing how excessive reasoning degrades performance (e.g., causing phrase and localization drift) on unambiguous queries, signaling that more thinking is detrimental.
Terminology used across episodes
This episode discusses
- To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation · Paper Radio
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- AURORA:Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation
- SAM 2: Segment Anything in Images and Videos
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
- Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation
- MAR3: Multi-Agent Recognition, Reasoning, and Reflection for Reference Audio-Visual Segmentation
- Adaptive Computation Time for Recurrent Neural Networks
- Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
- AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgeting
- CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction
- CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos
- VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learning
- Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation
The paper
To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation · Read on arXiv
IEIT SYSTEMS (Beijing) Co. · Tongji University
Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controller improves over downstream confidence routing and closely matches Short-then-Decide while using far fewer tokens. Proxy comparisons show that the onset representation contains predictive information not fully recovered by the evaluated hand-crafted features. Reusing this onset state for lightweight, single forward-pass routing reduces autoregressive token consumption by 60% while retaining about 96% of always-long performance. It further provides a cost-quality trade-off on R squared-AVSBench without target-label tuning, rather than an accuracy improvement over always-long. Error analysis links harmful long reasoning to phrase and localization drift, illustrating the benefit of bypassing unnecessary reasoning on simple queries. Ultimately, our results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "To Think or Not to Think".
Jane: Here's the gist of the paper. Multimodal large language models increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding <ref:2610.21613#pg6> The results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem <ref:2610.21613#pg5> Pruning redundant reasoning tokens Simple queries can be harmed by excessive reasoning, creating an “overthinking trap” associated with phrase and localization drift <ref:2610.21613#pg6> This happens when forcing the model through a lengthy reasoning chain on unambiguous inputs adds unsupported details, drifts from visual evidence, and ultimately produces a worse segmentation mask <ref:2610.21613#pg5>.
Tom: The paper shows that the value of reasoning is not a property of the model alone, but of the particular input-model interaction <ref:2610.21613#pg6>.
Jane: How the training signal is built The core idea involves examining the MLLM’s internal hidden state at generation onset, which they call honset <ref:2610.21613#pg6>.
Tom: This state is pre-decisional because it has already integrated video, audio, and text evidence but contains no generated reasoning content <ref:2610.21613#pg6>.
Title and authors: Tom: Let's look at the authors of this work. Haojie Hu and his team are from IEIT Systems in Beijing, Tongji University. They've been working on these multimodal models for a while, so they have deep experience with what those internal states look like.
Jane: It’s good to see a team that understands both the high-level reasoning ideas and the low-level engineering required to actually implement this kind of pre-decisional control.
Lu: The approach they’re taking here is to use something called honset, which is a hidden state in the model’s encoder right at the very beginning of generation, before any reasoning tokens even appear.
Meng: So they are essentially trying to capture all the video and audio and text evidence integrated into that state before it gets messy with actual thought processes.
Lalam: They define this generation-onset representation as the final layer state right after all the input tokens but before any autoregressive reasoning token is added, which they call honset.
Tom: And they found that this honset state actually encodes an informative signal about whether or not reasoning is needed for that specific task.
Jane: That means instead of just guessing based on some general rule, the model can look at the input and decide upfront if it should spend its reasoning budget.
The paper's summary: Tom: The main contribution they are making is providing this representation-level evidence that tells you whether using more reasoning will help or not, and they show that this prediction is made before the first token is generated.
Jane: So, the big result here is a linear classifier trained on this honset state can predict if additional reasoning will improve segmentation with seventy point one percent accuracy. That’s pretty strong against text-only features and uncertainty signals.
Lu: What really stands out to me is that they translate this finding into something practical: a single forward-pass controller that reads honset and dynamically assigns a zero, short, or long reasoning budget without needing any extra model passes.
Meng: That’s what I want to hear. If we can do this with just one forward pass and it cuts token consumption by sixty percent while keeping ninety-six percent of the performance you get from always-long reasoning, that’s something we can actually deploy.
Lalam: The system they designed is a lightweight controller that reads honset and decides whether to use zero, short, or long reasoning before segmentation begins.
Tom: That single forward pass controller is key because it means the decision is made immediately at generation onset without any external difficulty evaluators interfering with the process.
The paper's improvements: Jane: They also look at how they characterize that overthinking trap, and they show that forced chain-of-thought can actually degrade segmentation on simple inputs while wasting computation.
Tom: And they do this by matching reasoning budgets to outcomes, looking at how the phrase length and the Grounding DINO top one box intersection over union drops in those harmful long cases.
Lu: They also explore using other signals, like cross-modal attention entropy and latent contextual shift, to see if simple heuristics can govern budget selection.
Meng: But they found that those hand-crafted features don't actually capture all the predictive information that is available in that honset state they identified.
Lalam: And another thing they showed is that selecting the budget direction based on this onset controller is useful for the external readout or routing pipeline, meaning it helps guide what happens next in the system.
Tom: So, to summarize their improvements: they have a cost-aware oracle that selects the cheapest budget within a quality tolerance of zero point zero two for each sample.
Conclusion: Jane: So wrapping up this whole discussion on "To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation", the main implication is that we can make reasoning much more efficient by checking an internal state before committing to long reasoning.
Tom: The big win they show is a single forward pass controller that reads honset and dynamically assigns zero, short, or long reasoning budgets, which cuts token consumption by sixty percent while keeping ninety-six percent of the always-long performance.
Lu: It basically recasts adaptive Ref-AVS reasoning as a pre-decisional representation problem where the generation onset state predicts whether more reasoning is needed.
Meng: For deployment, this is huge because it means we can significantly reduce inference costs and latency without losing much quality if the query isn't too ambiguous.
Lalam: They found that this method reduces token consumption by thirty-nine point six percent while only dropping the J and F score by zero point zero zero seven relative to the short then decide approach.
Tom: It’s a really solid result because they show that the decision happens first on ninety-nine point eight percent of samples, which makes this controller very reliable.
Jane: It shows that we don't need to always be thinking deeply for every single query; we can be smarter about when to apply our computational power.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game