SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Project page: https://armenjeddi.github.io/scopd
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reasoning vision-language models (VLMs) process images and videos as long sequences of visual tokens, making inference expensive.
Terminology
Abstract
Reasoning vision-language models (VLMs) process images and videos as long sequences of visual tokens, making inference expensive. Training-free token pruning reduces this cost, but aggressive compression can sharply degrade performance, often attributed to irreversible loss of task-relevant visual information. We show that this explanation is incomplete. In a fixed-context Pass@K analysis, repeated sampling from the same pruned visual representation recovers many examples missed by greedy decoding, indicating that useful visual evidence can remain accessible but be used unreliably. We call this the representation-utilization gap. Motivated by this observation, we introduce SCOPD, a sparse-context on-policy self-distillation framework in which a student generates reasoning trajectories from pruned visual tokens while a privileged full-context teacher supervises the same on-policy prefixes. SCOPD requires no ground-truth responses, architectural changes, or additional inference-time computation. We further introduce SCOPD+, which uses a small visual-budget intervention to identify visually sensitive response positions and selectively distill them. At 10% visual-token retention, the Vanilla model retains 86.37% of its unpruned performance across 13 benchmarks. SCOPD raises this to 90.49%, while SCOPD+ further improves it to 92.43%. Across token budgets, benchmarks, and pruning operators, our results show that efficient reasoning depends not only on which visual information survives pruning, but also on how reliably the model learns to use it.
Sources
- Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Evaluating Large Language Models Trained on Code
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- Similarity-Aware Token Pruning: Your VLM but Faster
- AVIS: Adaptive Test-Time Scaling for Vision-Language Models
- Trajectory-Refined Distillation
- VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information
- Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
- Visual-Advantage On-Policy Distillation for Vision-Language Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Self-Distillation Enables Continual Learning
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation
- Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models