Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation
cs.CV, cs.AI, cs.CL
Submitted: 2026-05-12
Updated: 2026-11-06
Comments: Accepted to NeurIPS 2026
Project page: https://seonghoon-yu.github.io/Masking-KD-Page
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent think-answer approaches in VLMs, such as Qwen3-VL-Thinking, boost reasoning performance by leveraging intermediate thinking steps before the final answer…
Terminology
Abstract
Recent think-answer approaches in VLMs, such as Qwen3-VL-Thinking, boost reasoning performance by leveraging intermediate thinking steps before the final answer, but their computational cost becomes substantial, especially for larger VLMs. To distill such capabilities into compact think-answer VLMs, a primary objective is to improve the student's ability to utilize visual evidence throughout its reasoning trace, as long think-answer traces suffer from visual forgetting issues. To this end, we introduce a novel think-answer distillation framework that encourages the student to anchor its thinking on visual information by masking the student's salient reasoning prefixes. To compensate for such masked textual cues, the student is encouraged to rely more on visual evidence as an alternative source of information during distillation. Our masking strategies include: 1) token-wise salient reasoning-prefix masking, which masks high-influence reasoning prefixes selectively for each next-token prediction, and 2) self-paced masking budget scheduling, which gradually increases the masking scale according to distillation difficulty, measured by the discrepancy between teacher--student distributions. In the distillation phase, the student is guided by our salient reasoning-prefix mask, which blocks both future tokens and salient reasoning cues, in place of the standard causal mask used for auto-regressive language modeling. Experimental results show that our approach outperforms recent open-source VLMs, VLM distillation, and self-distillation methods on multimodal reasoning benchmarks, while further analyzes confirm enhanced visual utilization along the student thinking process.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- OpenAI o1 System Card
- Ovis: Structural Embedding Alignment for Multimodal Large Language Model
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces
- MiMo-VL Technical Report
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
- R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs