Visual Token Coding for Video Multimodal Large Language Models

arXiv:2608.28008 · cs.CV · Submitted 2026-08-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Visual Token Coding for Video Multimodal Large Language Models".

Title and authors: Tom: So, to recap, they’re using video compression ideas—like those from HEVC—to structure how we process video tokens for these massive multimodal models called MLLMs.

Jane: Basically, instead of treating every single frame as an equal piece of data, they organize the visual information into something like intra frames and predictive frames.

Tom: Exactly. The core mechanism is predicting what comes next and then looking at the difference between what you predicted and what actually happened to see if that difference is just noise or actual new information.

Lu: They call this residual coding, where small residuals mean the content is already covered by the reference frame, so you can compress it more aggressively.

Jane: And that’s why they call it Visual Token Coding, or VTC. It’s about treating the video like a compressed file so the model only has to pay attention to what actually changes over time.

Tom: But they took that baseline and built VTCDy, which adds dynamic resolution input, token allocation based on complexity, and spatial coverage top-k for better results.

Lu: The point of those extra parts is that they can handle very long videos—like a whole movie—with only fifty percent of the tokens needed.

Jane: I mean fifty percent! And they show that even when you cut the budget way down to twenty five percent, you still keep almost ninety eight point eight percent of the performance.

Tom: That’s a big retention rate for token pruning, especially on Qwen3-VL.

Meng: And from a deployment standpoint, they managed to reduce latency significantly, cutting it by two and a half times on an RTX three thousand nine hundred and ten graphics card when only using twenty five percent of the budget.

Lu: What this really changes is that you don't need massive fine-tuning just to make these models handle longer video sequences efficiently under tight token constraints.

Jane: It means the structure of video compression itself provides a practical way to save tokens for long-form multimodal understanding, instead of just guessing which tokens are important.

Tom: It moves the conversation from just picking important frames to using temporal residuals to expose redundancy at a whole video level.

Meng: And that structural organization is what makes VTCDy work better than just randomly removing tokens by importance scores.

Jane: That opens up a lot of possibilities for how we build these systems if we can apply that video coding structure across other complex data types.

Lu: And they also found that performance remained stable even when they changed the threshold for semantic I frames between one point two and two point zero.

Meng: That’s a good sign; it means the method isn't overly sensitive to tiny adjustments in how we define what counts as a reference frame.

Tom: It moves the focus from just picking important frames to using temporal residuals to expose redundancy at a whole video level.

Meng: And that structural organization is what makes VTCDy work better than just randomly removing tokens by importance scores.

Jane: That opens up a lot of possibilities for how we build these systems if we can apply that video coding structure across other complex data types.

The paper's summary: Tom: So we’re looking at how they actually made VTCDy better than the baseline VTC, and it comes down to these three specific dynamic mechanisms they introduced.

Jane: It basically says that just having a basic I and P frame structure isn't enough for really demanding video tasks.

Lu: Right. They added Dynamic Resolution Input, or DyRSO, which means they’re not sticking to one fixed resolution across the whole video timeline anymore.

Tom: So instead of using one resolution everywhere, the system uses a multi-resolution organization along the timeline, building a group of pictures that might use lower resolution frames where things are stable.

Jane: That helps with temporal coverage without just increasing the visual encoding cost proportionally.

Meng: So if you’re watching something mostly static, you don't need high-resolution data for every single second, which is smart when thinking about running models on limited hardware.

Tom: And then there’s Dynamic Token Allocation, or DyTA.

Lu: This part is inspired by how HEVC handles bit budgets; it distributes a set token budget across all the frames based on how complex or dependent those frames are on each other.

Jane: So if a frame is super busy with action, it gets more of that precious visual token budget, while a simpler frame gets less.

Tom: That seems like a very practical way to manage resource usage in real-time video processing without needing the model itself to be retrained for every scenario.

Meng: If you’re deploying this, that adaptive allocation sounds like it will make the inference much more efficient at scale.

Lu: And finally, Spatial Coverage Top-K, or SC-TopK. This is about organizing the entire grid of visual tokens into different regions and picking tokens from those regions to keep coverage in every area while prioritizing the ones that show big changes across those areas.

Jane: That’s a way to ensure you don't miss important local details while still capturing the overall motion happening at a broader scale.

Tom: And I have to say, when I look at the ablation study results for those three parts, DyRSO really helps with fine-grained detail and testing on things like LVBench.

Lu: But SC-TopK actually produced the strongest combined result overall when they looked at their MLVU average score.

Meng: So even though it’s more complex, that combined approach seems to give the best performance metric for this video token pruning technique.

Jane: It really shows that these three components aren't just one thing doing all the work; they complement each other nicely to give you better results.

Tom: And I think what this means is that structured prediction and residual coding can effectively expose redundancy at the video level, not just frame by frame.

The paper's improvements: Tom: So to wrap up, this paper on Visual Token Coding for Video Multimodal Large Language Models basically shows that you can use video compression principles to structure how these models process visual information into tokens.

Jane: It means they’ve found a way to organize video data like an I and P frame system so the AI only focuses on the actual changes between frames, which saves a ton of computational effort.

Lu: Exactly. The idea is that by modeling the video as a sequence of semantic reference frames and residual changes, you can prune tokens based on whether they are redundant or actually new information at that specific time point.

Tom: It’s about structured prediction overriding just simple frame-by-frame importance scores with this temporal context.

Jane: I mean for someone listening, it’s like teaching the AI to stop wasting its attention on things that haven't changed in a long time, which is crucial when dealing with those long videos we see every day.

Meng: From a deployment view, if this works as well as they say without needing massive model retraining, that drastically lowers the cost of running these multimodal systems.

Tom: I gotta ask, though—what does it actually cost to run this VTCDy system in production?

Lu: Well, they showed a latency reduction from four thousand one hundred nine milliseconds down to one thousand six hundred forty-four milliseconds on an RTX three thousand nine hundred and ten when only using twenty five percent of the budget.

Jane: That’s a two point five times speedup, which is a big deal for real-time or near real-time applications.

Lalam: Yes, I'm sure about this application of residual coding to video tokens. It feels like the most direct way to handle the high dimensionality of visual data efficiently without sacrificing too much fidelity on complex scenes.

Lu: They also noted that the ablation study showed that DyRSO and SC-TopK work together best for getting the highest score on their MLVU average, which is a solid piece of evidence for their combined approach.

Jane: So, to summarize, this paper suggests that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets.

Tom: It's about using dynamic resolution and temporal budgeting to achieve superior compression without needing additional model tuning on the core architecture.

Lu: And it’s a big step because it moves us toward better representation learning that understands video structure, not just pixel importance.

Meng: If they can keep this level of performance retention when cutting the budget in half, that opens up much larger applications for these models.

Jane: The limitation they mentioned is mostly related to the complexity of implementing those dynamic allocation and coverage mechanisms perfectly at scale, but the paper provides a clear path forward for how structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level.

Tom: So, for anyone working on long-video MLLMs right now, this VTCDy approach seems like it gives you a solid, implementable framework that doesn't require additional model tuning.

Meng: It’s a structured prediction and residual coding idea applied to the visual token space.

Jane: I think what this paper is doing is showing that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets.

Conclusion: Tom: So we're looking at this paper Visual Token Coding for Video Multimodal Large Language Models now. It comes from Chenxin Fang and the team at Xiamen University, and they’re proposing a new method called VTC.

Jane: What does that actually mean for us in practical terms?

Tom: Basically, it means they're trying to figure out how to handle the massive amount of video data efficiently using a token budget.

Lu: The core idea is organizing the video into I and P frames, which is a structure from video compression.

Jane: So an I frame holds the main scene information, and the P frames only hold what changed since that reference frame.

Tom: And they use those differences between what they predicted versus what actually happened to estimate token redundancy in terms of tokens.

Meng: But they don't stop at just that baseline VTC; they build VTCDy, which is Visual Token Coding with Dynamic enhancements.

Jane: What are these new dynamic parts that make it better?

Tom: They introduce Dynamic Resolution Input, or DyRSO, which uses a multi-resolution Group of Pictures or GOP along the video timeline.

Lu: That means they can handle different resolutions depending on where you are in the video, expanding the temporal coverage without increasing visual encoding cost.

Jane: And then there's Dynamic Token Allocation, or DyTA.

Tom: Then they have Spatial Coverage Top-K, or SC-TopK. This part divides the entire token grid into regions to make sure they keep coverage in each area while prioritizing tokens that show large changes across those areas.

Lu: It's about balancing local detail with global motion information in the selection process.

Jane: So, if I’m driving and I need to understand a long sequence of video, this VTCDy approach lets the model focus its attention where it matters most temporally and spatially.

Tom: The results they show are pretty strong; VTCDy achieves an average performance retention of one point zero zero one percent when you use fifty percent of the token budget for Qwen3-VL.

Meng: And they still keep ninety seven point eight percent average performance even when you cut that token budget down to twenty five percent.

Jane: That’s quite a jump, keeping nearly ninety eight point eight percent accuracy even with a quarter of the tokens.

Lu: It suggests that this structured compression method is genuinely applicable to existing multimodal models without needing massive retraining or tuning on the model itself.

Jane: So, the qualitative analysis they did shows how this allocation works on a video timeline; for example, assigning about seven point seven percent to what’s almost a duplicate of previous content and nearly thirty percent to substantial content change.

Jane: It’s interesting how those three components—resolution, budget allocation, and spatial coverage—play complementary roles; they aren't just one thing doing all the heavy lifting.

Tom: And that adaptive allocation is especially important when the budget gets more restrictive, where you need to be smart about how you spend every single token.

Lu: What this really changes is that you don't need massive fine-tuning just to make these models handle longer video sequences efficiently under tight token constraints.

Jane: It means the structure of video compression itself provides a practical way to save tokens for long-form multimodal understanding, instead of just

Xiamen University

cs.CV

Submitted: 2026-08-28

Updated: 2026-11-18

Comments: 11 pages, 5 figures

Code: https://github.com/Msr233/VTC

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC)…

Key concepts

Visual Token Coding (VTC)
This paradigm organizes visual tokens like video frames into semantic I-frames (for scene references) and P-frames (for changes). By comparing the current frame to a predicted reference, VTC identifies redundant information, allowing for targeted token pruning based on temporal residuals.
Dynamic Resolution Input (DyRSO)
Instead of using all high-resolution frames, DyRSO organizes the video into a multi-resolution Group of Pictures (GOP). This replaces some full-resolution input with lower-resolution source frames, effectively expanding the video's temporal coverage while keeping the visual encoding cost manageable.
Dynamic Token Allocation (DyTA)
Inspired by HEVC rate control, DyTA distributes a limited token budget across different frames. It allocates more tokens to frames that exhibit higher coding complexity or significant temporal changes, ensuring critical information is prioritized over less important ones.
Spatial Coverage Top-K (SC-TopK)
This technique divides the visual token grid into distinct regions. It selects tokens by prioritizing those in regions showing large changes while ensuring regional coverage remains intact. This balances capturing significant motion with maintaining a complete view of the scene.

Terminology

Summary

In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC), which is inspired by classical video coding principles and shows that structured compression is applicable to token pruning The experimental results show that VTCDy achieves an average performance retention of 100. p. 0

Where the paper says it

cn Abstract In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.

1% with a 50% token budget for Qwen3-VL, while still retaining 97. p. 0
Where the paper says it

The experimental results show that VTCDy achieves an average performance retention of 100. 1% with a 50% token budget for Qwen3-VL, while still retaining 97.

8% of the average performance when the token budget is reduced to 25%

Visual Token Coding Paradigm

The principle of VTC is inspired by traditional video coding, such as HEVC (Sullivan et al. 2012) As shown in Fig. p. 0

Where the paper says it

Concretely, the principle of VTC is inspired by traditional video coding, such as HEVC (Sullivan et al. As shown in Fig.

1, instead of transmitting all RGB frames, video coding organizes them into Intra-coded frames and Predictive-coded frames, denoted as I and P frames, respectively An I-frame retains nearly complete visual information and is then used as a reference to structurally compress P frames through motion-compensated prediction (Sullivan et al. p. 0
Where the paper says it

1, an I-frame retains nearly complete visual information and is then used as a reference to structurally compress P frames through motion-compensated prediction (Sullivan et al. During decoding, RGB information can be restored through motion compensation and residual addition using motion vectors and prediction residuals.

2012) The paper aims to apply the principle of video coding to token compression and achieve a general visual token coding (VTC) paradigm for existing MLLMs

Enhancements to VTC

Building on the baseline VTC, we further introduce three novel designs: Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), to further improve the compression ratio and performance retention for advanced video MLLMs, and we term this new approach VTCDy Compared with traditional video coding, our VTCDy performs token-level compression and can be directly applied to existing MLLMs

Dynamic Budgeting Mechanisms

VTCDy extends the baseline by incorporating dynamic resolution input, dynamic token allocation, and spatial coverage top-K to address limitations of fixed-resolution input and uniform frame-level budgets Dynamic Resolution Input (DyRSO) expands temporal coverage by building a multi-resolution Group of Pictures (GOP) along the video timeline Dynamic Token Allocation (DyTA) is inspired by frame-level HEVC rate control, distributing a finite bit budget across frames according to coding dependencies and complexity Spatial Coverage Top-K (SC-TopK) divides the token grid into regions to maintain intra-frame regional coverage while prioritizing large changes

Experimental Validation and Results

To validate the proposed VTC baseline and VTCDy, we apply them to three video MLLMs, namely Qwen3-VL-8B (Bai et al. 2025b), LLaVA-OneVision (Li et al. p. 1

Where the paper says it

To validate the proposed VTC baseline and VTCDy, we apply them to three video MLLMs, namely Qwen3-VL-8B (Bai et al. 2025b), LLaVA-OneVision (Li et al.

2024) and LLaVA-OneVision-2 (An et al. p. 1
Where the paper says it

2025b), LLaVA-OneVision (Li et al. 2024) and LLaVA-OneVision-2 (An et al.

2026), and also compare them with the SOTA token compression methods on a set of video benchmarks The experimental results show that VTCDy achieves performance competitive with or better than SOTA methods, e. p. 1
Where the paper says it

3% and 101. 6% of the LLaVA-OneVision performance with a 50% token retention ratio, but also show that our VTCDy achieves performance competitive with or better than SOTA methods, e.

g., 1.4 percentage points higher average performance than FlashVID on Qwen3-VL

Conclusion

We propose VTC, a video-coding-inspired framework that organizes visual tokens as semantic I/P frames and allocates them by temporal residuals VTCDy further introduces dynamic resolution, temporal budgeting, and spatial coverage to achieve superior compression results without additional model tuning These results indicate that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets

Pruning redundant reasoning tokens

VTC goes beyond token importance estimation and emphasizes whether visual information can be predicted from existing content or is newly introduced at the current temporal position By transferring reference prediction and residual coding to the visual feature space, VTC constructs semantic I/P frames, where I frames preserve scene references and P frames represent changes relative to these references

How the training signal is built

For Ft ∈ RN×D, VTC adopts identity prediction using the full feature at the preceding temporal position: Fbt = Fref VTC then computes the semantic residual and its token-level energy by Et,i = ∥Rt(i)∥2, where Rt = Ft − Fbt In Eq. 8, small residuals indicate that the video content is already represented by the reference and can therefore be compressed more aggressively

Where the gains come from

The ablation study shows that DyRSO mainly improves fine-grained detail and LVBench, DyTA benefits multi-detail and holistic understanding, and SC-TopK produces the strongest combined result The combined score achieves the strongest MLVU average The gain is larger at the more restrictive budget, where adaptive allocation is more important than uniform token removal Moreover, VTCDy reduces latency from 4109 to 1644 ms (2.50×) on one RTX 3090 at 25% retention

Overall Summary

VTCDy computes the mean bitcost µt and the 90th percentile bitcost pt for each causal interval and obtains the following complexity score: qt = αµet + (1 − α)pet This combination reflects both sustained coding costs and short-term complex temporal changes The final results confirm that VTCDy achieves the best performance on most benchmarks

The qualitative analysis visualizes how VTCDy redistributes tokens along a video timeline, assigning 7.7% to a near duplicate and 29. p. 6

Where the paper says it

Input Frames VTC VTC Near-duplicate - very few tokens Related content - moderate tokens Scene cut - new semantic I frame 100. 8% 𝑫𝒚 Figure 4: Token allocation under the same budget.

6% to substantial content change It also detects the scene cut as a new semantic I frame The masks show that this allocation preserves changing content while suppressing stable redundancy

The ablation study shows that resolution, temporal budgeting, and spatial coverage play complementary roles The MLVU average rises from 67.3 for VTC to 68. p. 0

Where the paper says it

cn, {pengjun, zhouyiyi, rrji}@xmu. cn Abstract In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC).

p. 0
Where the paper says it

cn, {pengjun, zhouyiyi, rrji}@xmu. cn Abstract In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC).

2 with DyRSO, 68. p. 0
Where the paper says it

, HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach V T CDy.

p. 0
Where the paper says it

, HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach V T CDy.

9 with DyTA, and 69. p. 0
Where the paper says it

, HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach V T CDy.

p. 0
Where the paper says it

, HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach V T CDy.

2 with SC-TopK The second block varies the semantic I-frame threshold τ. p. 6
Where the paper says it

2 with SC-TopK; LVBench improves from 40. The second block varies the semantic I-frame threshold τ .

p. 6
Where the paper says it

2 with SC-TopK; LVBench improves from 40. The second block varies the semantic I-frame threshold τ .

Performance is stable from 1. p. 6
Where the paper says it

The second block varies the semantic I-frame threshold τ . Performance is stable from 1.

p. 6
Where the paper says it

The second block varies the semantic I-frame threshold τ . Performance is stable from 1.

2 to 2.0 despite the change in I-frame frequency P90 performs best on LVBench, while the combined score achieves the strongest MLVU average The gain is larger at the more restrictive budget, where adaptive allocation is more important than uniform token removal

Structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level VTCDy further introduces dynamic resolution, temporal budgeting, and spatial coverage to achieve superior compression results without additional model tuning More broadly, structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level

VTCDy computes the mean bitcost µt and the 90th percentile bitcost pt for each causal interval and obtains the following complexity score: qt = αµet + (1 − α)pet This combination reflects both sustained coding costs and short-term complex temporal changes The final results confirm that VTCDy achieves the best performance on most benchmarks

VTCDy further introduces dynamic resolution, temporal budgeting, and spatial coverage to achieve superior compression results without additional model tuning These results indicate that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets More broadly, structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level

The ablation study shows that resolution, temporal budgeting, and spatial coverage play complementary roles The MLVU average rises from 67.3 for VTC to 68.2 with DyRSO, 68.9 with DyTA, and 69.2 with SC-TopK The second block varies the semantic I-frame threshold τ. Performance is stable from 1.2 to 2.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Visual Token Coding (VTC) allows for token compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. This enables MLLMs to process longer videos with a 50% token budget while retaining performance comparable to full models.

  2. VTCDy extends VTC with Dynamic Resolution Input (DyRSO), which uses a multi-resolution temporal organization that replaces part of the full-resolution input with low-resolution source frames, increasing temporal coverage without proportional increases in visual encoding cost.

  3. VTCDy incorporates Dynamic Token Allocation (DyTA), which uses HEVC bitcost to allocate the visual token budget across frames according to coding dependencies and complexity, ensuring that more complex temporal positions receive larger budgets.

  4. VTCDy includes Spatial Coverage Top-K (SC-TopK), which divides the token grid into regions to select tokens, preserving regional coverage while prioritizing large changes, as it preserves regional coverage while prioritizing large changes.

Abstract

In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this base paradigm, we further extend VTC with a set of dynamic budgeting designs towards more cost-effective token coding, termed VTC dy. To validate VTC and VTC dy, we apply them to three MLLMs and conduct experiments on a set of video understanding benchmarks. The experimental results show that as a base paradigm, VTC can achieve an average performance retention of 98.4% on Qwen3-VL under a 50% token compression ratio, and the enhanced VTC dy can further improve the retention to 100.3%, outperforming a set of compared methods. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is given in the supplementary materials.

Sources

Related papers