Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

arXiv:2608.28138 · cs.CV · Submitted 2026-08-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models".

Jane: The gist The Token-Budget Distillation framework proposes a parameter-efficient fine-tuning method that transfers full-token semantics to compressed video vision-language models by using a dual-path teacher and student design under a…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this paper today, "Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models." It sounds like they're tackling a big problem where adapting these video models is too expensive because of the sheer number of visual tokens they process across frames.

Jane: Exactly. The title suggests they are figuring out how to move the meaning from a model that sees everything to one that only sees a small part, all while keeping it accurate. It’s about making sure you don't lose the fine details when you heavily compress the video input.

Lu: The authors are Guo, Luo, Zhang, Wang, and Wang. They come from institutions like Sun Yat-sen University and UBC. They have a solid background in representation learning and training these kinds of large models that process video data.

Meng: From my side, what I’m interested in is the efficiency part. If you can adapt these massive video models with just a limited token budget, that means less computational cost for fine-tuning and inference at scale, which is something we need when deploying things to production.

Lalam: I'm picking up on the idea of "transferring full-token semantics," which suggests they aren't just trying to make a compressed model; they’re trying to preserve the actual high-level understanding that the original full model had.

Tom: So, what is this Token-Budget Distillation framework actually proposing? Basically, it sets up a dual-path system where there’s a teacher branch and a student branch. The teacher processes everything without compression and LoRA adapters off, acting like the original full-token model.

Jane: And the student branch is forced to operate under this strict token budget, using some kind of compressed representation. They use this setup to make sure the student learns how to handle those scarce visual tokens correctly.

Lu: The core idea is that by having that robust teacher branch providing guidance, you compel the student to reconstruct that high-fidelity behavior even when it only has a limited visual budget. This moves away from just simple compression tuning toward a method focused on preserving semantics.

Meng: I worry about the complexity of running both branches at once, but if it works, it means we can fine-tune models efficiently without losing the core reasoning capabilities that come from seeing all the frames.

Lalam: The paper says they formulate this by minimizing an objective function that balances standard task loss with a distillation objective which includes answer-region KL alignment and ground-truth-anchored margin distillation.

Tom: So, what are the specific improvements they introduce to handle that semantic drift we talked about earlier? They use two main distillation techniques to fix it. First, answer-region KL alignment, which filters out noise by only mimicking the teacher’s most confident token distributions.

Jane: And second, ground-truth-anchored margin distillation. That one is supposed to enforce a gap between the target response and any competing tokens so the student doesn't just get fuzzy answers.

Lu: Those mechanisms are designed to pull the student’s conceptual embedding back toward the teacher’s manifold and ensure it mimics those decisive boundary gaps, not just a soft distribution.

Meng: If they can keep attention stable under severe token reduction, that would be huge for deploying these models on resource-constrained hardware. It addresses a real issue at scale where performance degrades quickly when you cut down the input size by ninety percent.

Lalam: The paper shows that this approach successfully stabilizes the transfer and even keeps the student’s attention map consistent with the teacher’s, which means it successfully allocates its limited token budget to exactly the same salient regions identified by that teacher.

Tom: So, to wrap up, what does this mean for us? It frames compressing models not just as a way to save tokens but as a semantic recovery problem guided by the teacher. They show that this dual-path architecture works across different video VLMs like LLaVA-Video and Qwen3-VL-8B-Instruct.

Jane: It really suggests that when you have limited resources, the adaptation process needs to be more structured than just trying to fit a model to the data. You need that teacher guidance to prevent losing critical reasoning skills during compression.

Lu: The finding is that they can achieve high performance even under aggressive compression regimes, preserving ninety-seven point zero percent of the Vanilla model’s average accuracy on LLaVA-Video when the retention ratio is ten percent. That's a strong empirical result for their method of transferring full-token semantics to compressed video vision-language models.

Meng: I just have one reservation: they admit that this requires running a full-token teacher during the adaptation phase, which means it still demands high training-time memory, and the student’s performance is always capped by the irreversible information loss of whatever compression module you choose.

Lalam: They are right about those limitations; the framework demonstrates that effective adaptation under token compression is about teacher-guided semantics recovery rather than just plain task fitting.

The paper's summary: Tom: So, to recap, this paper is about taking these massive video models and adapting them when you have to drastically cut down on visual tokens for efficiency. They’re proposing this dual-path system—a full-token teacher and a compressed student—to keep the original model's deep understanding intact while only using a small budget.

Jane: Right, so the core idea is transferring all that high-level meaning from the big model to a tiny one without losing it. It’s not just about making something faster; it’s about preserving what makes the AI smart in the first place when you compress things down.

Lu: The researchers are focused on how they use this structure to prevent "semantic drift," which is when those compressed models start making weird, generalized guesses instead of precise actions. They introduce two specific distillation methods to fight that: answer-region KL alignment and ground-truth-anchored margin distillation.

Meng: I need to know what the numbers are here because if it’s just a qualitative description, it doesn't matter for production. What’s the actual trade-off? Does this method keep ninety percent of the performance or does it lose something crucial in that compression?

Lalam: The authors show that under aggressive compression, like ten percent retention ratio, they maintain ninety-seven percent of the Vanilla model's average accuracy on LLaVA-Video. That’s a solid number for preserving semantics.

Tom: Ninety-seven percent is high, but I’m allergic to claims without context. The paper shows that the mechanism works across different video backbones like LLaVA-Video and Qwen3-VL-8B-Instruct, which suggests this isn't just a specific trick for one architecture.

Jane: It really is a versatile framework, which is important because most of these methods are tailored to just one model structure. The paper also does an ablation study showing exactly how much each piece—the KL alignment versus the margin distillation—contributes to that stability.

Lu: They show that the attention drift analysis confirms something key: the student successfully allocates its severely limited token budget to the exact same salient visual regions that the teacher identified, even when it’s only using a fraction of those tokens. That means visual grounding is preserved.

Meng: Preserving grounding under such heavy compression is what I care about most for deployment. But they do admit a limitation: you still have to run that full-token teacher during the adaptation phase, which means you still have high memory demands for training time, and the performance ceiling is still limited by whatever compression module you choose.

Lalam: So it’s not a perfect solution yet; it’s a guided recovery method. It reframes adaptation under compression as fixing semantics rather than just fitting the task data.

Tom: Exactly. The implication for me is that we can now explore adapting these video VLMs more aggressively in terms of input size without completely sacrificing the fine-grained reasoning capabilities they were trained on originally, provided we use this teacher guidance structure.

Jane: And for listeners who don't read papers, it means that when you compress a model to save space, you need a smart way to tell it *what* visual details are most important so it doesn't just hallucinate or ignore the crucial parts of the video.

Lu: The future work they mention suggests exploring how this teacher-guided concept can be generalized beyond just video inputs, perhaps applying similar distillation principles to other complex multimodal tasks.

Meng: I’m watching that for potential applications in real-time systems where memory and speed are both major constraints. It moves the goalposts on what’s feasible to deploy right now.

The paper's improvements: Tom: So, to summarize the improvements, they aren't just about having two paths; they’re about using specific distillation constraints to make sure that student branch actually learns what matters from the teacher’s guidance. They introduce answer-region KL alignment and GT-anchored margin distillation for that purpose.

Jane: That sounds a bit technical—KL alignment and margin distillation—but I get it as a way to filter out the noise, you know? If you’re trying to teach someone something important, you don't want them mimicking every single word; you want them mimicking the most confident, relevant parts.

Lu: Exactly. The KL alignment is meant to only let the student mimic the teacher’s most confident token distributions and pull its conceptual embedding back onto the teacher’s manifold. It filters out all that extra noise caused by compression.

Meng: So it's a quality control layer on top of just having a teacher branch, ensuring the student isn't just copying everything but only what is semantically decisive for the task at hand.

Tom: And then you’ve got the ground-truth-anchored margin distillation, which enforces a robust confidence gap between the target response and any competing tokens. This means if there are two possible answers, it forces the student to learn that difference in confidence.

Jane: That makes sense for reliability. It stops the student from giving a soft answer when it should be making a firm decision based on visual evidence. It’s about enforcing sharpness in the decision boundary.

Lalam: The result they show empirically is that this combination of constraints leads to better separation between the correct answer and its distractors, which confirms that the distilled student model makes fewer ranking errors.

Tom: I like seeing a clear improvement in those margins, specifically going from a margin of two point two three two five to over two point seven eight three two when they apply these distillation techniques. That’s tangible evidence of better decision-making.

Lu: It also stabilizes the attention map dramatically, which means the student successfully targets its limited token budget to exactly the same salient regions that the teacher identifies, even though it’s using ninety percent fewer tokens. That’s a big win for visual grounding preservation.

Jane: So what does this mean for us? It means that when we compress video models for efficiency, we can use these specific distillation rules to keep them from losing their ability to "see" the important stuff in the video.

Meng: I still see that full-token teacher requirement as a headache for training time, though. Running a full-token model repeatedly during adaptation is computationally expensive and doesn't scale well for our production pipelines right now.

Tom: That’s the caveat they admitted—the memory demand for the teacher branch is high during fine-tuning. But the trade-off seems to be worth it if you need that level of semantic fidelity under extreme compression.

Lu: The paper suggests this framework could be applied more broadly, not just to video, but maybe to other multimodal models where token budgets are tight and preserving complex semantics is essential.

Conclusion: Tom: So we’re wrapping up this discussion on "Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models." Basically, they showed that you can adapt these big video models under a token budget by using a teacher and student setup with specific distillation rules.

Jane: That really boils down to the idea that we can transfer the full meaning from a huge model into something much smaller without losing those critical fine details when we compress the input data.

Lu: The authors prove that this dual-path design, combined with answer-region KL alignment and ground-truth margin distillation, keeps the student focused on what’s actually important visually. It stabilizes the representation reconstruction significantly.

Meng: I still think running that full teacher during training is a big hurdle for deployment because of the memory cost, even if we get ninety-seven percent accuracy on benchmarks like LLaVA-Video at ten percent retention ratio.

Lalam: I’m confident in my assessment; the paper shows that this method works across different video backbones, proving it's a general approach for transferring semantics under compression.

Tom: It really frames adaptation under heavy compression not just as a way to save tokens but as a problem of recovering deep semantics guided by the teacher.

Jane: It means that when you’re building these models, you need structured guidance from the original model to handle limited resources effectively instead of just hoping the compression works out fine.

Lu: The limitation they flag is that it still depends on your choice of compression module because whatever compression you pick introduces irreversible information loss. That's a real constraint we have to keep in mind.

Meng: So, it’s an efficient path forward, but it doesn't solve the fundamental issue of losing data during the initial compression step itself.

Lalam: I think so; this is a powerful tool for when you need high fidelity under strict constraints, but it’s not a magic fix for every compression scenario.

Sun Yat-sen University · The University of British Columbia

cs.CV

Submitted: 2026-08-28

Updated: 2026-10-19

Comments: ACM MM 2026. Code is available at https://github.com/Seele77/TBD

Code: https://github.com/Seele77/TBD

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist The Token-Budget Distillation framework proposes a parameter-efficient fine-tuning method that transfers full-token semantics to compressed video vision-language models by using a dual-path

Key concepts

Token Budgeted Setting: This constraint limits a video sample to using only a fixed number of visual tokens. The goal is to train models that can perform complex tasks without consuming an excessively large amount of input tokens, which is crucial for handling long videos efficiently.
Dual-Path Teacher and Student Design: The framework uses two parallel pathways from the same base model: a teacher branch that runs with full capability (no compression) and a student branch that operates under token constraints. The teacher provides an exact, high-quality target for the student to learn from, guiding its adaptation process.
Answer-Region KL Alignment: This distillation technique forces the compressed student to mimic only the most confident and relevant parts of the teacher's output distribution. It filters out noise by aligning the student's conceptual embeddings with those of the teacher, ensuring it focuses on high-quality semantic regions.
GT-Anchored Margin Distillation: This method enforces a clear confidence gap between the model's predicted answer and incorrect competing tokens. By preserving these decisive boundary gaps, the student learns to distinguish correct answers sharply from distractors, improving its discriminative power.

Terminology

Distillation — 1× in this paper · explained in 2 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist The Token-Budget Distillation framework proposes a parameter-efficient fine-tuning method that transfers full-token semantics to compressed video vision-language models by using a dual-path teacher and student design under a fixed token budget.

Problem Setup

The paper identifies the core challenge as the computational expense of adapting video vision-language models due to token explosion across frames, making both fine-tuning and inference costly The fundamental challenge lies in preserving the finegrained visual semantics critical for accurate reasoning when operating on this heavily compressed representation To mitigate this bottleneck, we consider a token-budgeted setting where the model is constrained to consume at most B visual tokens per video sample. The objective is formulated as: Φ∗ = arg min Φ ∑ (v,q,y) ∈D L y;v, q, z f, z c; Θ, Φ (2).

Token-Budget Distillation Framework

Token-Budget Distillation (TBD) is a dual-path training framework that maps full-token semantics into a compressed-token student. For each training instance (v, q, y), TBD constructs two distinct forward pathways from the identical pretrained backbone: a teacher branch processing z f and a student branch processing z c. The teacher branch operates with compression disabled and LoRA adapters deactivated, effectively acting as an exact proxy for the original full-token model. This asymmetric architecture ensures that the teacher branch provides a robust, anchor-like full-token target, compelling the student to reconstruct this high-fidelity behavior from scarce visual evidence.

Reliable Distillation Under Token Compression

To combat semantic drift caused by token compression, TBD introduces reliable answer-region distillation. Answer-region KL alignment ensures the student only mimics the teacher’s most confident and relevant token distributions, filtering out noise and pulling the student’s conceptual embedding back to the teacher’s manifold. Simultaneously, GT-anchored margin distillation enforces a robust confidence gap between the target response and competing tokens. This ensures that the compressed student mimics not just the softened distribution, but the teacher’s decisive boundary gaps.

Overall Objective and Optimization

The complete optimization objective balances standard autoregressive training with our reliability-aware distillation. The primary task loss ensures fidelity to the downstream instruction: LCE = − 1 A ∑ i∈A log sigma(l s i)y i (8). The distillation objective aggregates the KL and margin terms: LKD = LKL + λm Lmargin (9). The final training loss is dynamically scaled: L = λceLCE + ˆlambdakd (t) LKD (10).

Empirical Success and Analysis

TBD consistently outperforms compression-only baselines under both moderate and aggressive compression regimes. On LLaVA-Video at an aggressive R = 10% retention ratio, TBD preserves 97.0% of the Vanilla model’s average accuracy. The ablation study confirms that the proposed mechanisms are complementary: answerregion KL distillation aligns output distributions, GT-anchored margin distillation preserves discrimination, and reliability-aware filtering stabilizes noisy signals. Furthermore, attention drift analysis shows that the TBD student successfully allocates its severely limited token budget to the exact same salient regions identified by the teacher.

Conclusion and Limitations

TBD reframes token-compressed adaptation as teacher-guided semantics recovery rather than plain task fitting. The framework demonstrates that effective adaptation under token compression is not solely an efficiency challenge, but a semantics preservation problem. A limitation noted is that TBD requires executing a full-token teacher during the adaptation phase, which maintains high training-time memory demands. Additionally, the student’s performance upper bound is inherently constrained by the irreversible information loss of the chosen training-free compression module.

--- Page 1 ---

Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models. Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, and Wenhao Wang. 2026. Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3835088

--- Page 2 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, and Wenhao Wang. LLaVA-Video [61], LLaVA-OneVision [19], and Qwen3-VL-8B-Instruct [2].

--- Page 3 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Figure 1: Overview of the Token-Budget Distillation (TBD) framework. A full-token teacher branch provides reliable guidance, while a compressed student branch with LoRA is optimized under a fixed token budget.

--- Page 4 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Algorithm 1 Token-Budget Distillation (TBD) Optimization Input: Training set D, frozen VLM backbone Θ, trainable LoRA adapters Φ, compression budget B, KD hyperparameters (lambdace, λm, λkd). Output: Optimized student LoRA parameters Φ∗

--- Page 5 ---

MM ’26, November 10–14-2026. LLaVA-Video [61], LLaVA-OneVision [19], and Qwen3-VL-8B-Instruct [2].

--- Page 6 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 1: Main results on four video understanding benchmarks under different token retention ratios. We compare TBD with the uncompressed Vanilla model and representative token reduction baselines on three backbones: LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct.

--- Page 7 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 2: Ablation study on LLaVA-Video at retention ratio R = 10%.

--- Page 8 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 3: Effect of different compression modules within TBD on LLaVA-Video at retention ratio R = 10%.

--- Page 9 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 4: Effect of Compression Modules.

--- Page 10 ---

MM ’26, November 10–14-2026. Figure 3: Decision boundary analysis on 200 randomly sampled VideoMME instances for LLaVA-Video at R = 10%.

--- Page 11 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Figure 4: Attention drift visualization on a randomly selected VideoMME example for LLaVA-Video at retention ratio R = 10%.

--- Page 12 ---

MM ’26, November 10–14-2026. Figure 2: Logit distribution alignment on 100 randomly sampled VideoMME instances for LLaVA-Video at R = 10%.

--- Page 13 ---

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Figure 3: Decision boundary analysis on 200 randomly sampled VideoMME instances for LLaVA-Video at R = 10%.

--- Page 14 ---

MM ’26, November 10–14-2026. Figure 4: Attention drift visualization on a randomly selected VideoMME example for LLaVA-Video at retention ratio R = 10%.

--- Page 15 ---

MM ’26, November 10–14-2026. Figure 3: Decision boundary analysis on 200 randomly sampled VideoMME instances for LLaVA-Video at R = 10%.

--- Page 16 ---

MM ’26, November 10–14-2026. Figure 4: Attention drift visualization on a randomly selected VideoMME example for LLaVA-Video at retention ratio R = 10%.

--- Page 17 ---

MM ’26, November 10–14-2026.

Improvements for AI systems

textbfTeacher-Guided Semantic Preservation in Compressed VLMs: The TBD Framework enables transferring full-token semantics to compressed video vision-language models. This allows video VLMs to adapt efficiently under a strict token budget while preventing semantic drift by employing a dual-path teacher-student design with specific distillation constraints like answer-region KL distillation and GT-anchored margin distillation.
textbfMechanistic Rehabilitation of Internal Representations: TBD addresses the pathology where the student’s visual attention drifts from precise, discriminative actions (e.g., 'Riding a bicycle') towards ambiguous, generalized conceptual representations (e.g., 'Moving object'). This is achieved by ensuring the student reconstruct[s] this high-fidelity behavior from scarce visual evidence through constraints that enforce robust confidence gap[s] between the target response and competing tokens.
textbfUniversal Applicability Across VLM Architectures: The TBD framework demonstrates its effectiveness across diverse backbones, as it is evaluated on LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct. This confirms that the parameter-efficient fine-tuning framework is not tied to a specific encoder but can be plugged into existing video VLMs while maintaining high performance under aggressive compression regimes."
textbf Enhanced Decision Boundary Sharpness: The integration of GT-anchored margin distillation results in a student model that explicitly shifts rightward (left panel) in decision space, leading to an increase in the mean margin from 2.2325 to 2.7832 and a reduction in the negative-margin ratio, which confirms the distilled student better separates the correct answer from the hardest distractor and makes fewer ranking errors.
textbf Robust Attention Focus Under Compression: The system successfully arrests semantic drift by demonstrating that the TBD-adapted student’s attention map remains highly consistent with that of the teacher, meaning it successfully allocates its severely limited token budget to the exact same salient regions identified by the teacher despite discarding 90% of tokens, ensuring visual grounding is preserved."

Abstract

Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.