Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
The study investigated how removing visual tokens from multimodal models affects their confidence calibration. It found that using moderate coverage-based pruning improves calibration at unchanged accuracy, proving the selection rule is more important than token budget alone. Attention-based selection maintains confidence even as accuracy drops.