Visual Token Coding for Video Multimodal Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Visual Token Coding for Video Multimodal Large Language Models".
Title and authors: Tom: So, to recap, they’re using video compression ideas—like those from HEVC—to structure how we process video tokens for these massive multimodal models called MLLMs.
Jane: Basically, instead of treating every single frame as an equal piece of data, they organize the visual information into something like intra frames and predictive frames.
Tom: Exactly. The core mechanism is predicting what comes next and then looking at the difference between what you predicted and what actually happened to see if that difference is just noise or actual new information.
Lu: They call this residual coding, where small residuals mean the content is already covered by the reference frame, so you can compress it more aggressively.
Jane: And that’s why they call it Visual Token Coding, or VTC. It’s about treating the video like a compressed file so the model only has to pay attention to what actually changes over time.
Tom: But they took that baseline and built VTCDy, which adds dynamic resolution input, token allocation based on complexity, and spatial coverage top-k for better results.
Lu: The point of those extra parts is that they can handle very long videos—like a whole movie—with only fifty percent of the tokens needed.
Jane: I mean fifty percent! And they show that even when you cut the budget way down to twenty five percent, you still keep almost ninety eight point eight percent of the performance.
Tom: That’s a big retention rate for token pruning, especially on Qwen3-VL.
Meng: And from a deployment standpoint, they managed to reduce latency significantly, cutting it by two and a half times on an RTX three thousand nine hundred and ten graphics card when only using twenty five percent of the budget.
Lu: What this really changes is that you don't need massive fine-tuning just to make these models handle longer video sequences efficiently under tight token constraints.
Jane: It means the structure of video compression itself provides a practical way to save tokens for long-form multimodal understanding, instead of just guessing which tokens are important.
Tom: It moves the conversation from just picking important frames to using temporal residuals to expose redundancy at a whole video level.
Meng: And that structural organization is what makes VTCDy work better than just randomly removing tokens by importance scores.
Jane: That opens up a lot of possibilities for how we build these systems if we can apply that video coding structure across other complex data types.
Lu: And they also found that performance remained stable even when they changed the threshold for semantic I frames between one point two and two point zero.
Meng: That’s a good sign; it means the method isn't overly sensitive to tiny adjustments in how we define what counts as a reference frame.
Tom: It moves the focus from just picking important frames to using temporal residuals to expose redundancy at a whole video level.
Meng: And that structural organization is what makes VTCDy work better than just randomly removing tokens by importance scores.
Jane: That opens up a lot of possibilities for how we build these systems if we can apply that video coding structure across other complex data types.
The paper's summary: Tom: So we’re looking at how they actually made VTCDy better than the baseline VTC, and it comes down to these three specific dynamic mechanisms they introduced.
Jane: It basically says that just having a basic I and P frame structure isn't enough for really demanding video tasks.
Lu: Right. They added Dynamic Resolution Input, or DyRSO, which means they’re not sticking to one fixed resolution across the whole video timeline anymore.
Tom: So instead of using one resolution everywhere, the system uses a multi-resolution organization along the timeline, building a group of pictures that might use lower resolution frames where things are stable.
Jane: That helps with temporal coverage without just increasing the visual encoding cost proportionally.
Meng: So if you’re watching something mostly static, you don't need high-resolution data for every single second, which is smart when thinking about running models on limited hardware.
Tom: And then there’s Dynamic Token Allocation, or DyTA.
Lu: This part is inspired by how HEVC handles bit budgets; it distributes a set token budget across all the frames based on how complex or dependent those frames are on each other.
Jane: So if a frame is super busy with action, it gets more of that precious visual token budget, while a simpler frame gets less.
Tom: That seems like a very practical way to manage resource usage in real-time video processing without needing the model itself to be retrained for every scenario.
Meng: If you’re deploying this, that adaptive allocation sounds like it will make the inference much more efficient at scale.
Lu: And finally, Spatial Coverage Top-K, or SC-TopK. This is about organizing the entire grid of visual tokens into different regions and picking tokens from those regions to keep coverage in every area while prioritizing the ones that show big changes across those areas.
Jane: That’s a way to ensure you don't miss important local details while still capturing the overall motion happening at a broader scale.
Tom: And I have to say, when I look at the ablation study results for those three parts, DyRSO really helps with fine-grained detail and testing on things like LVBench.
Lu: But SC-TopK actually produced the strongest combined result overall when they looked at their MLVU average score.
Meng: So even though it’s more complex, that combined approach seems to give the best performance metric for this video token pruning technique.
Jane: It really shows that these three components aren't just one thing doing all the work; they complement each other nicely to give you better results.
Tom: And I think what this means is that structured prediction and residual coding can effectively expose redundancy at the video level, not just frame by frame.
The paper's improvements: Tom: So to wrap up, this paper on Visual Token Coding for Video Multimodal Large Language Models basically shows that you can use video compression principles to structure how these models process visual information into tokens.
Jane: It means they’ve found a way to organize video data like an I and P frame system so the AI only focuses on the actual changes between frames, which saves a ton of computational effort.
Lu: Exactly. The idea is that by modeling the video as a sequence of semantic reference frames and residual changes, you can prune tokens based on whether they are redundant or actually new information at that specific time point.
Tom: It’s about structured prediction overriding just simple frame-by-frame importance scores with this temporal context.
Jane: I mean for someone listening, it’s like teaching the AI to stop wasting its attention on things that haven't changed in a long time, which is crucial when dealing with those long videos we see every day.
Meng: From a deployment view, if this works as well as they say without needing massive model retraining, that drastically lowers the cost of running these multimodal systems.
Tom: I gotta ask, though—what does it actually cost to run this VTCDy system in production?
Lu: Well, they showed a latency reduction from four thousand one hundred nine milliseconds down to one thousand six hundred forty-four milliseconds on an RTX three thousand nine hundred and ten when only using twenty five percent of the budget.
Jane: That’s a two point five times speedup, which is a big deal for real-time or near real-time applications.
Lalam: Yes, I'm sure about this application of residual coding to video tokens. It feels like the most direct way to handle the high dimensionality of visual data efficiently without sacrificing too much fidelity on complex scenes.
Lu: They also noted that the ablation study showed that DyRSO and SC-TopK work together best for getting the highest score on their MLVU average, which is a solid piece of evidence for their combined approach.
Jane: So, to summarize, this paper suggests that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets.
Tom: It's about using dynamic resolution and temporal budgeting to achieve superior compression without needing additional model tuning on the core architecture.
Lu: And it’s a big step because it moves us toward better representation learning that understands video structure, not just pixel importance.
Meng: If they can keep this level of performance retention when cutting the budget in half, that opens up much larger applications for these models.
Jane: The limitation they mentioned is mostly related to the complexity of implementing those dynamic allocation and coverage mechanisms perfectly at scale, but the paper provides a clear path forward for how structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level.
Tom: So, for anyone working on long-video MLLMs right now, this VTCDy approach seems like it gives you a solid, implementable framework that doesn't require additional model tuning.
Meng: It’s a structured prediction and residual coding idea applied to the visual token space.
Jane: I think what this paper is doing is showing that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets.
Conclusion: Tom: So we're looking at this paper Visual Token Coding for Video Multimodal Large Language Models now. It comes from Chenxin Fang and the team at Xiamen University, and they’re proposing a new method called VTC.
Jane: What does that actually mean for us in practical terms?
Tom: Basically, it means they're trying to figure out how to handle the massive amount of video data efficiently using a token budget.
Lu: The core idea is organizing the video into I and P frames, which is a structure from video compression.
Jane: So an I frame holds the main scene information, and the P frames only hold what changed since that reference frame.
Tom: And they use those differences between what they predicted versus what actually happened to estimate token redundancy in terms of tokens.
Meng: But they don't stop at just that baseline VTC; they build VTCDy, which is Visual Token Coding with Dynamic enhancements.
Jane: What are these new dynamic parts that make it better?
Tom: They introduce Dynamic Resolution Input, or DyRSO, which uses a multi-resolution Group of Pictures or GOP along the video timeline.
Lu: That means they can handle different resolutions depending on where you are in the video, expanding the temporal coverage without increasing visual encoding cost.
Jane: And then there's Dynamic Token Allocation, or DyTA.
Tom: Then they have Spatial Coverage Top-K, or SC-TopK. This part divides the entire token grid into regions to make sure they keep coverage in each area while prioritizing tokens that show large changes across those areas.
Lu: It's about balancing local detail with global motion information in the selection process.
Jane: So, if I’m driving and I need to understand a long sequence of video, this VTCDy approach lets the model focus its attention where it matters most temporally and spatially.
Tom: The results they show are pretty strong; VTCDy achieves an average performance retention of one point zero zero one percent when you use fifty percent of the token budget for Qwen3-VL.
Meng: And they still keep ninety seven point eight percent average performance even when you cut that token budget down to twenty five percent.
Jane: That’s quite a jump, keeping nearly ninety eight point eight percent accuracy even with a quarter of the tokens.
Lu: It suggests that this structured compression method is genuinely applicable to existing multimodal models without needing massive retraining or tuning on the model itself.
Jane: So, the qualitative analysis they did shows how this allocation works on a video timeline; for example, assigning about seven point seven percent to what’s almost a duplicate of previous content and nearly thirty percent to substantial content change.
Jane: It’s interesting how those three components—resolution, budget allocation, and spatial coverage—play complementary roles; they aren't just one thing doing all the heavy lifting.
Tom: And that adaptive allocation is especially important when the budget gets more restrictive, where you need to be smart about how you spend every single token.
Lu: What this really changes is that you don't need massive fine-tuning just to make these models handle longer video sequences efficiently under tight token constraints.
Jane: It means the structure of video compression itself provides a practical way to save tokens for long-form multimodal understanding, instead of just
Xiamen University
cs.CV
Submitted: 2026-08-28
Updated: 2026-11-18
Comments: 11 pages, 5 figures
Code: https://github.com/Msr233/VTC
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC)…
Key concepts
- Visual Token Coding (VTC)
- This paradigm organizes visual tokens like video frames into semantic I-frames (for scene references) and P-frames (for changes). By comparing the current frame to a predicted reference, VTC identifies redundant information, allowing for targeted token pruning based on temporal residuals.
- Dynamic Resolution Input (DyRSO)
- Instead of using all high-resolution frames, DyRSO organizes the video into a multi-resolution Group of Pictures (GOP). This replaces some full-resolution input with lower-resolution source frames, effectively expanding the video's temporal coverage while keeping the visual encoding cost manageable.
- Dynamic Token Allocation (DyTA)
- Inspired by HEVC rate control, DyTA distributes a limited token budget across different frames. It allocates more tokens to frames that exhibit higher coding complexity or significant temporal changes, ensuring critical information is prioritized over less important ones.
- Spatial Coverage Top-K (SC-TopK)
- This technique divides the visual token grid into distinct regions. It selects tokens by prioritizing those in regions showing large changes while ensuring regional coverage remains intact. This balances capturing significant motion with maintaining a complete view of the scene.
Terminology
Summary
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC), which is inspired by classical video coding principles and shows that structured compression is applicable to token pruning The experimental results show that VTCDy achieves an average performance retention of 100. p. 0 cn
Abstract
In this paper, we propose a new token compression paradigm
for video Multimodal Large Language Models (MLLMs),
termed Visual Token Coding (VTC). Inspired by classical
video coding principles, e. The experimental results show that
VTCDy achieves an average performance retention of 100. 1%
with a 50% token budget for Qwen3-VL, while still retaining
97.Where the paper says it
Where the paper says it
Visual Token Coding Paradigm
The principle of VTC is inspired by traditional video coding, such as HEVC (Sullivan et al. 2012) As shown in Fig. p. 0 Concretely, the principle of VTC is inspired
by traditional video coding, such as HEVC (Sullivan et al. As shown in Fig. 1, an I-frame retains nearly
complete visual information and is then used as a reference to
structurally compress P frames through motion-compensated
prediction (Sullivan et al. During decoding, RGB information can be restored through motion compensation and
residual addition using motion vectors and prediction residuals.Where the paper says it
Where the paper says it
Enhancements to VTC
Building on the baseline VTC, we further introduce three novel designs: Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), to further improve the compression ratio and performance retention for advanced video MLLMs, and we term this new approach VTCDy Compared with traditional video coding, our VTCDy performs token-level compression and can be directly applied to existing MLLMs
Dynamic Budgeting Mechanisms
VTCDy extends the baseline by incorporating dynamic resolution input, dynamic token allocation, and spatial coverage top-K to address limitations of fixed-resolution input and uniform frame-level budgets Dynamic Resolution Input (DyRSO) expands temporal coverage by building a multi-resolution Group of Pictures (GOP) along the video timeline Dynamic Token Allocation (DyTA) is inspired by frame-level HEVC rate control, distributing a finite bit budget across frames according to coding dependencies and complexity Spatial Coverage Top-K (SC-TopK) divides the token grid into regions to maintain intra-frame regional coverage while prioritizing large changes
Experimental Validation and Results
To validate the proposed VTC baseline and VTCDy, we apply them to three video MLLMs, namely Qwen3-VL-8B (Bai et al. 2025b), LLaVA-OneVision (Li et al. p. 1 To validate the proposed VTC baseline and VTCDy, we
apply them to three video MLLMs, namely Qwen3-VL-8B
(Bai et al. 2025b), LLaVA-OneVision (Li et al. 2025b), LLaVA-OneVision (Li et al. 2024) and
LLaVA-OneVision-2 (An et al. 3% and
101. 6% of the LLaVA-OneVision performance with a 50%
token retention ratio, but also show that our VTCDy achieves
performance competitive with or better than SOTA methods,
e.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
We propose VTC, a video-coding-inspired framework that organizes visual tokens as semantic I/P frames and allocates them by temporal residuals VTCDy further introduces dynamic resolution, temporal budgeting, and spatial coverage to achieve superior compression results without additional model tuning These results indicate that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets
Pruning redundant reasoning tokens
VTC goes beyond token importance estimation and emphasizes whether visual information can be predicted from existing content or is newly introduced at the current temporal position By transferring reference prediction and residual coding to the visual feature space, VTC constructs semantic I/P frames, where I frames preserve scene references and P frames represent changes relative to these references
How the training signal is built
For Ft ∈ RN×D, VTC adopts identity prediction using the full feature at the preceding temporal position: Fbt = Fref VTC then computes the semantic residual and its token-level energy by Et,i = ∥Rt(i)∥2, where Rt = Ft − Fbt In Eq. 8, small residuals indicate that the video content is already represented by the reference and can therefore be compressed more aggressively
Where the gains come from
The ablation study shows that DyRSO mainly improves fine-grained detail and LVBench, DyTA benefits multi-detail and holistic understanding, and SC-TopK produces the strongest combined result The combined score achieves the strongest MLVU average The gain is larger at the more restrictive budget, where adaptive allocation is more important than uniform token removal Moreover, VTCDy reduces latency from 4109 to 1644 ms (2.50×) on one RTX 3090 at 25% retention
Overall Summary
VTCDy computes the mean bitcost µt and the 90th percentile bitcost pt for each causal interval and obtains the following complexity score: qt = αµet + (1 − α)pet This combination reflects both sustained coding costs and short-term complex temporal changes The final results confirm that VTCDy achieves the best performance on most benchmarks
The qualitative analysis visualizes how VTCDy redistributes tokens along a video timeline, assigning 7.7% to a near duplicate and 29. p. 6 Input
Frames
VTC
VTC
Near-duplicate - very few tokens Related content - moderate tokens Scene cut - new semantic I frame
100. 8%
𝑫𝒚
Figure 4: Token allocation under the same budget.Where the paper says it
The ablation study shows that resolution, temporal budgeting, and spatial coverage play complementary roles The MLVU average rises from 67.3 for VTC to 68. p. 0 cn,
{pengjun, zhouyiyi, rrji}@xmu. cn
Abstract
In this paper, we propose a new token compression paradigm
for video Multimodal Large Language Models (MLLMs),
termed Visual Token Coding (VTC). cn,
{pengjun, zhouyiyi, rrji}@xmu. cn
Abstract
In this paper, we propose a new token compression paradigm
for video Multimodal Large Language Models (MLLMs),
termed Visual Token Coding (VTC). , HEVC, VTC performs structured compression by predicting the I/P frames of a video
and measuring their frame-wise residuals to estimate token
redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation
(DyTA), and Spatial Coverage Top-K (SC-TopK), and term
this new approach V T CDy. , HEVC, VTC performs structured compression by predicting the I/P frames of a video
and measuring their frame-wise residuals to estimate token
redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation
(DyTA), and Spatial Coverage Top-K (SC-TopK), and term
this new approach V T CDy. , HEVC, VTC performs structured compression by predicting the I/P frames of a video
and measuring their frame-wise residuals to estimate token
redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation
(DyTA), and Spatial Coverage Top-K (SC-TopK), and term
this new approach V T CDy. , HEVC, VTC performs structured compression by predicting the I/P frames of a video
and measuring their frame-wise residuals to estimate token
redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation
(DyTA), and Spatial Coverage Top-K (SC-TopK), and term
this new approach V T CDy. 2 with SC-TopK; LVBench
improves from 40. The second block varies the semantic I-frame threshold
τ . 2 with SC-TopK; LVBench
improves from 40. The second block varies the semantic I-frame threshold
τ . The second block varies the semantic I-frame threshold
τ . Performance is stable from 1. The second block varies the semantic I-frame threshold
τ . Performance is stable from 1.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level VTCDy further introduces dynamic resolution, temporal budgeting, and spatial coverage to achieve superior compression results without additional model tuning More broadly, structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level
VTCDy computes the mean bitcost µt and the 90th percentile bitcost pt for each causal interval and obtains the following complexity score: qt = αµet + (1 − α)pet This combination reflects both sustained coding costs and short-term complex temporal changes The final results confirm that VTCDy achieves the best performance on most benchmarks
VTCDy further introduces dynamic resolution, temporal budgeting, and spatial coverage to achieve superior compression results without additional model tuning These results indicate that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets More broadly, structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level
The ablation study shows that resolution, temporal budgeting, and spatial coverage play complementary roles The MLVU average rises from 67.3 for VTC to 68.2 with DyRSO, 68.9 with DyTA, and 69.2 with SC-TopK The second block varies the semantic I-frame threshold τ. Performance is stable from 1.2 to 2.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Visual Token Coding (VTC) allows for token compression by
predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy.
This enables MLLMs to process longer videos with a50% token budget
while retaining performance comparable to full models. -
VTCDy extends VTC with Dynamic Resolution Input (DyRSO), which uses
a multi-resolution temporal organization that replaces part of the full-resolution input with low-resolution source frames,
increasing temporal coverage without proportional increases in visual encoding cost. -
VTCDy incorporates Dynamic Token Allocation (DyTA), which uses
HEVC bitcost to allocate the visual token budget across frames according to coding dependencies and complexity,
ensuring thatmore complex temporal positions receive larger budgets.
-
VTCDy includes Spatial Coverage Top-K (SC-TopK), which divides the token grid into regions to select tokens, preserving regional coverage while prioritizing large changes, as it
preserves regional coverage while prioritizing large changes.
Abstract
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this base paradigm, we further extend VTC with a set of dynamic budgeting designs towards more cost-effective token coding, termed VTC dy. To validate VTC and VTC dy, we apply them to three MLLMs and conduct experiments on a set of video understanding benchmarks. The experimental results show that as a base paradigm, VTC can achieve an average performance retention of 98.4% on Qwen3-VL under a 50% token compression ratio, and the enhanced VTC dy can further improve the retention to 100.3%, outperforming a set of compared methods. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is given in the supplementary materials.
Sources
- LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
- Qwen2.5-VL Technical Report
- Qwen3-VL Technical Report
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
- AdaCodec: A Predictive Visual Code for Video MLLMs
- Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
- LLaVA-OneVision: Easy Visual Task Transfer
- VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
- Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processing
- CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
- LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
- PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
- ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
- Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling
- Long Context Transfer from Language to Vision
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs