Not All Visual Tokens Are Equally Safe to Remove:Consequence-Sensitive Visual Token Compression

This paper introduces consequence-sensitive visual token compression, a new formulation for allocating visual computation in vision-language models (VLMs) based on the downstream cost of errors rather than solely on content-derived signals like attention, redundancy, or uncertainty.

The authors argue that existing visual token compression methods (e.g., FastV, SparseVLM, VisionZip, PyramidDrop) optimize for preserving average accuracy while reducing the number of processed visual tokens, which implicitly assumes that all errors have equal consequences on downstream tasks. However, in real-world deployments such as document, chart, and infographic understanding, "misreading an invoice amount, transaction identifier, or medical value can trigger different consequences than misreading a section title or a decorative attribute, so the cost of an error is inherently asymmetric across samples. Importantly, such asymmetry is often available before inference through request-level information, such as question type, task identity, or application workflow."

The proposed framework follows a calibrate-then-allocate procedure:

1. Offline calibration: Estimate tier-specific error–budget curves ê t(b) on a calibration split by evaluating each candidate budget on a predefined grid. Select the optimal allocation by minimizing the estimated cost-weighted objective: (B hi, B lo) = arg min J(B hi, B lo), subject to a fixed total budget constraint.

2. Online inference: Each query is assigned to a consequence tier using an inference-time-observable signal (e.g., a frozen keyword rule over question text, or task identity). The corresponding tier budget is then applied.

The optimization problem is formalized as: minimize Σ n t c t e t(b t) subject to Σ n t b t ≤ B, where c t is the consequence weight, n t is tier size, and e t(b) is the tier-level error-vs-budget curve.

Two budget realization mechanisms are used: (a) true token deletion (encode at full resolution, rank tokens with a fixed selector, delete the rest exactly) and (b) resolution reallocation (resize the image so the encoder emits approximately the budgeted token count).

On a controlled within-task benchmark (DocVQA) where high- and low-consequence questions share the same document image distribution, consequence-sensitive allocation reduces high-stakes errors from 0.300 to 0.133 under the same total token budget (McNemar 54:4, p≈3×10−12), a 2.25× reduction. The content-driven allocator performs no better than uniform allocation (4:4, p=1.0) because visual content is identical across tiers. The trade-off is explicit: descriptive-tier error increases to 0.843, resulting in worse unweighted accuracy (0.488 vs. 0.337 for uniform), demonstrating the method optimizes cost-weighted error rather than average accuracy.

The two tiers exhibit statistically indistinguishable token sensitivity (central-difference marginals at the uniform split: 0.00173 vs. 0.00171, slope SE ≈ 2×10−4). The optimal allocation depends on the consequence ratio r:

• At r=1 (symmetric costs), uniform allocation is optimal.

• At r=3, an interior optimum exists.

• At r≥5, the optimum moves toward the boundary, transferring more tokens to high-consequence samples.

The paper derives an allocation frontier: uniform allocation is optimal when errors are equally costly, and token transfer toward high-consequence questions becomes increasingly beneficial as the cost gap grows.

The allocation principle generalizes across:

• Three dense benchmarks: DocVQA, ChartQA, InfographicVQA (break-even cost ratios of 2.8, 3.6, and 0.7 respectively)

• Two budget realization mechanisms: token deletion and resolution reallocation

• Two VLM architectures: Qwen2.5-VL-7B and LLaVA-OneVision-7B (break-even ratio 2.79, nearly identical to Qwen's 2.82)

• Multiple token selection strategies: redundancy-based, random, saliency, FastV, SparseVLM, VisionZip

On a realistic three-task mixture (DocVQA/TextVQA/VQAv2 with task-identity tiers), tier-based allocation cuts cost-weighted error by 38% (∆CWE CI [0.080, 0.188]) while achieving approximately 21% lower latency than full-resolution inference.

• Consequence signal validation: The keyword rule is validated against gold-answer content (90.3% of high-tier questions have quantitative answers vs. 32% of low-tier, κ=0.58) and an independent LLM judge (72.5% agreement, κ=0.45). A lightweight TF-IDF logistic regression classifier predicts tiers with 94.3% accuracy.

• Metric robustness: Results hold under strict exact match and official ANLS@0.5 metrics across all datasets.

• Split-half calibration: Predicted optima achieve rank 1/13 at r∈ 1,10,20 and rank 3/13 at r∈ 2,5 on held-out data.

The paper concludes that visual token compression should be evaluated not only by how much accuracy it preserves, but also by which errors it prevents. The framework is "lightweight and deployment-oriented, inferring consequence tiers from question or task information, calibrating tier-specific error–budget curves offline, and realizing the resulting budgets through token deletion or resolution reallocation."

Improvements to AI Systems:

1. Cost-aware visual token allocation: Implement a tiered token budget system where the VLM dynamically allocates more visual tokens to high-consequence queries (e.g., extracting invoice amounts, medical values) and fewer to low-consequence ones (e.g., descriptive attributes). This reduces critical errors by up to 2.25× under the same total compute budget, without requiring architectural changes.

2. Task-identity-based resource scheduling: For mixed workloads (e.g., document QA, chart QA, natural image QA), automatically assign consequence weights per task type at inference time (e.g., 5× for financial documents, 1× for casual image queries). This cuts cost-weighted error by 38% while reducing latency by 21% compared to full-resolution processing.

3. Offline calibration for optimal budget frontiers: Precompute tier-specific error-vs-budget curves on a calibration set. At deployment, use these curves to select the optimal token split for any given consequence ratio and total budget constraint, ensuring the system operates on the Pareto frontier between accuracy and compute.

4. Robust consequence-tier prediction: Integrate a lightweight TF-IDF logistic regression classifier (94.3% accuracy) or keyword rules to classify incoming queries into consequence tiers in real time, using only question text or task metadata—no extra inference cost.

5. Selector-agnostic compression: Apply consequence-sensitive allocation on top of any existing token selection method (FastV, SparseVLM, VisionZip, random, saliency). This makes the improvement a drop-in module for current VLM pipelines, preserving their strengths while adding cost-awareness.

6. Adaptive resolution reallocation: Instead of hard token deletion, resize input images so the encoder emits exactly the budgeted token count per tier. This enables the same cost-aware framework to work with resolution-based VLMs, reducing memory and compute without post-hoc token ranking.

What the Improved AI System Can Do:

• In a document-understanding API, it can prioritize accurate extraction of monetary values, dates, and IDs (high-consequence) over decorative text, reducing critical misreads by 2.25× at the same compute cost.

• In a mixed-task assistant (e.g., answering questions about charts, photos, and scanned forms), it automatically shifts compute toward high-stakes tasks (e.g., medical charts) while keeping latency low for casual queries, achieving 38% lower cost-weighted error and 21% faster responses.

• It can be deployed with any existing VLM (Qwen2.5-VL, LLaVA-OneVision) and any token selection strategy, requiring only a one-time offline calibration step and a simple runtime tier classifier—no retraining or architecture modification needed.

• It provides a principled trade-off knob: operators can set consequence ratios (e.g., 5:1 for high vs. low stakes) to match their application’s risk profile, and the system automatically finds the optimal token allocation.

Visual token compression for vision--language models (VLMs) has largely relied on criteria such as attention, redundancy, and uncertainty to maximize average accuracy under a fixed compute budget, implicitly assuming that all errors carry equal cost. However, the consequence of an incorrect prediction on downstream tasks is rarely symmetric: misreading an invoice amount can be far more costly than misclassifying a background color. Motivated by this, we introduce consequence-sensitive visual token compression, which allocates visual computation across requests according to their potential error costs. Our method follows a calibrate-then-allocate procedure, estimating consequence-specific error-budget curves offline and applying the calibrated token budgets online using consequence signals available from question or task information. On a controlled within-task benchmark, high- and low-consequence questions are drawn from the same document images, so content alone cannot reveal which questions are costly to get wrong. In this setting, our method reduces high-stakes errors from 0.300 to 0.133 under the same total token budget, whereas a content-driven allocator performs no better than uniform allocation. Measuring how error rates change with token budget across different cost ratios, we derive an allocation frontier: uniform allocation is optimal when errors are equally costly, and token transfer toward high-consequence questions becomes increasingly beneficial as the cost gap grows. This allocation principle generalizes well across three dense vision-language benchmarks, two budget realization mechanisms (token deletion and resolution reallocation), two VLM architectures, and multiple token selection strategies. On a realistic mixed workload, consequence-sensitive allocation reduces cost-weighted error by 38% while achieving approximately 21% lower latency than full-resolution inference.